feat: Revamp CrossValidationReport to use EstimatorReport #1091

glemaitre · 2025-01-10T18:50:27Z

closes #731
closes #860
closes #938
closes #978
closes #976
closes #977
closes #953
closes #1012
closes #1014

It is a revamp from the current CrossValidationReporter. I am working in another module such that:

CrossValidationReporter is still available such that we do the transition for the UI part
move CrossValidationReport to a private module since we want to encourage people to import either from skore or skore.sklearn.

There is quick demo here: https://gist.github.com/glemaitre/1216d2098e3f92e045fb064b9647f3f0

TODO

Address change parameter name in cross_validation_reporter #1012
Needs a lot of new tests.

…rt_3

github-actions · 2025-01-10T18:55:02Z

Documentation preview @ d46199e

github-actions · 2025-01-10T18:55:51Z

Coverage Report for backend

File	Stmts	Miss	Cover	Missing
venv/lib/python3.12/site-packages/skore
__init__.py	14	0	100%
__main__.py	8	1	80%	19
exceptions.py	3	0	100%
venv/lib/python3.12/site-packages/skore/cli
__init__.py	5	0	100%
cli.py	33	3	85%	104, 111, 117
color_format.py	43	3	90%	35–>40, 41–43
launch_dashboard.py	26	15	39%	36–57
quickstart_command.py	14	7	50%	37–51
venv/lib/python3.12/site-packages/skore/item
__init__.py	21	1	91%	46
cross_validation_item.py	137	10	93%	27–42, 373
item.py	42	13	69%	91, 94, 98–118
item_repository.py	65	5	92%	12–13, 211–212, 235
media_item.py	70	4	94%	15–18
numpy_array_item.py	25	1	93%	15
pandas_dataframe_item.py	34	1	95%	15
pandas_series_item.py	34	1	95%	15
polars_dataframe_item.py	32	1	94%	15
polars_series_item.py	27	1	94%	15
primitive_item.py	27	2	92%	13–15
sklearn_base_estimator_item.py	33	1	95%	15
skrub_table_report_item.py	10	1	86%	11
venv/lib/python3.12/site-packages/skore/persistence
__init__.py	0	0	100%
abstract_storage.py	22	1	95%	130
disk_cache_storage.py	33	1	95%	44
in_memory_storage.py	20	0	100%
venv/lib/python3.12/site-packages/skore/project
__init__.py	3	0	100%
create.py	52	8	88%	116–122, 132–133, 140–141
load.py	23	3	89%	43–45
open.py	14	0	100%
project.py	68	3	94%	118, 152, 156
venv/lib/python3.12/site-packages/skore/sklearn
__init__.py	5	0	100%
_base.py	140	4	97%	91, 94, 168–>173, 183–184
find_ml_task.py	45	1	96%	71–>87, 86
types.py	2	0	100%
venv/lib/python3.12/site-packages/skore/sklearn/_cross_validation
__init__.py	10	0	100%
metrics_accessor.py	163	0	100%
report.py	87	0	100%
venv/lib/python3.12/site-packages/skore/sklearn/_estimator
__init__.py	10	0	100%
metrics_accessor.py	265	4	97%	149–158, 183–>236, 191, 434–>437, 783–>786
report.py	121	0	99%	216–>222, 224–>226
utils.py	11	11	0%	1–19
venv/lib/python3.12/site-packages/skore/sklearn/_plot
__init__.py	4	0	100%
precision_recall_curve.py	121	1	98%	229–>246, 317
prediction_error.py	97	0	100%
roc_curve.py	128	0	100%
utils.py	84	0	100%
venv/lib/python3.12/site-packages/skore/sklearn/cross_validation
__init__.py	2	0	100%
cross_validation_helpers.py	47	4	90%	104–>136, 123–126
cross_validation_reporter.py	35	1	95%	177
venv/lib/python3.12/site-packages/skore/sklearn/cross_validation/plots
__init__.py	0	0	100%
compare_scores_plot.py	29	1	92%	10, 45–>48
timing_plot.py	29	1	94%	10
venv/lib/python3.12/site-packages/skore/sklearn/train_test_split
__init__.py	0	0	100%
train_test_split.py	36	2	94%	16–17
venv/lib/python3.12/site-packages/skore/sklearn/train_test_split/warning
__init__.py	8	0	100%
high_class_imbalance_too_few_examples_warning.py	17	3	78%	16–18, 80
high_class_imbalance_warning.py	18	2	88%	16–18
random_state_unset_warning.py	11	1	87%	15
shuffle_true_warning.py	9	0	91%	44–>exit
stratify_is_set_warning.py	11	1	87%	15
time_based_column_warning.py	22	1	89%	17, 69–>exit
train_test_split_warning.py	5	1	80%	21
venv/lib/python3.12/site-packages/skore/ui
__init__.py	0	0	100%
app.py	25	5	71%	24, 53–58
dependencies.py	7	1	86%	12
project_routes.py	61	0	100%
venv/lib/python3.12/site-packages/skore/utils
__init__.py	0	0	100%
_accessor.py	7	0	100%
_logger.py	21	4	84%	14–18
_patch.py	11	5	46%	19–35
_progress_bar.py	28	0	100%
_show_versions.py	31	0	100%
venv/lib/python3.12/site-packages/skore/view
__init__.py	0	0	100%
view.py	5	0	100%
view_repository.py	16	2	83%	8–9
TOTAL	2722	143	94%

Tests	Skipped	Failures	Errors	Time
573	3 💤	0 ❌	0 🔥	1m 16s ⏱️

skore/src/skore/utils/_progress_bar.py

augustebaum · 2025-01-13T10:32:46Z

Side-note: Was it intentional to link to PR 538?

glemaitre · 2025-01-13T11:45:26Z

Side-note: Was it intentional to link to PR 538?

Nop, it seems wrong. There might be a PR in the 500'ish to be close then :)

skore/src/skore/sklearn/_cross_validation/report.py

augustebaum

Apart from the caching question I just have a few suggestions.
Also, I guess the feature where we save intermediate data (e.g. save after evaluating each split) is pushed back to later?

skore/src/skore/sklearn/_cross_validation/__init__.py

skore/src/skore/sklearn/_cross_validation/report.py

skore/src/skore/sklearn/_estimator/__init__.py

skore/src/skore/sklearn/_estimator/report.py

Co-authored-by: Sylvain Combettes <[email protected]>

glemaitre · 2025-01-17T14:22:39Z

Also, I guess the feature where we save intermediate data (e.g. save after evaluating each split) is pushed back to later?

So here, the data are stored in the estimator_report that is a generator, so we should be on the path. If the idea, is to make an update on the reporter shown in the activity feed, we would an interuption when consuming the reporter and trigger partial computation of the view. But we have all the primitive to do it I think

Co-authored-by: Auguste Baum <[email protected]>

…l-ai#1143)

Closes probabl-ai#1110

Closes probabl-ai#1003 When running skore from source, it occurs that some contributors where puzzled by the nonexistence of the UI. This PR introduces a 404 page if the UI as not been built. ## preview ![Screenshot 2025-01-17 at 09 48 00](https://github.com/user-attachments/assets/0fb38493-31fe-4c27-b7e7-3ca83f5ce838)

Closes probabl-ai#1065 ![Screenshot 2025-01-17 at 14 58 49](https://github.com/user-attachments/assets/9a261b51-5d1f-4bb5-a383-8cf79d94c47f)

…i#1141) Co-authored-by: Thomas S. <[email protected]>

github-actions · 2025-01-17T14:29:44Z

Coverage Report for frontend

Status	Category	Percentage	Covered / Total
🔵	Lines	85.49%	3577 / 4184
🔵	Statements	85.49%	3577 / 4184
🔵	Functions	44.85%	61 / 136
🔵	Branches	79.15%	205 / 259

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
skore-ui/src/components/ProjectViewCard.vue	100%	100%	100%	100%
skore-ui/src/components/TreeAccordionItem.vue	83.96%	90%	20%	83.96%	23-27, 39-49, 59-60

Generated in workflow #94 for commit 25bb9ba by the Vitest Coverage Report Action

glemaitre · 2025-01-17T14:38:20Z

The comment should be addressed @augustebaum @sylvaincom

.github/workflows/backend.yml

…rt_3

augustebaum

Many thanks

…rt (#1156) Build upon #1091 We should merge #1091 to make it easy to review. This PR revisits the documentation for the `EstimatorReport` and `CrossValidationReport`. The idea is to provide different examples for different purposes: - technical section: it is a section to explain more in details some internal for user that might be interested. I think that we should move those inside this specific section because we should have more a pure data science section to only show the added values of our tools on data science - use-case section: it is indeed the section to show case our tools on real data. Here we want to show the added value on the data science side. --------- Co-authored-by: Sylvain Combettes <[email protected]> Co-authored-by: Auguste Baum <[email protected]> Co-authored-by: Matt J. <[email protected]> Co-authored-by: Thomas S. <[email protected]> Co-authored-by: MarieS-WiMLDS <[email protected]>

feat: Revamp CrossValidationReport to use EstimatorReport

76d72c6

github-actions bot assigned glemaitre Jan 10, 2025

Merge remote-tracking branch 'origin/main' into cross_validation_repo…

98d94d7

…rt_3

glemaitre marked this pull request as draft January 10, 2025 19:22

glemaitre added 20 commits January 11, 2025 00:57

add progress bar

8353b00

iter

b695666

make sure to use shared memory

4f8aa1c

fix progress bar glitches

5a42157

fix attribute sorting

e4accc8

iter

5ace5be

add metric accessor class

c6333ca

iter

82f64cb

add custom metric

b47786e

iter

fe110ff

roc curve

35e01ea

iter

4d0f16f

iter

d46199e

some cache optimization

ae11e23

hash computation optimization

e3a8a59

provide a way to send pos_label

68755aa

iter

3175047

iter

0256166

iter

8e6a532

more parallelism

2635f56

augustebaum reviewed Jan 13, 2025

View reviewed changes

skore/src/skore/utils/_progress_bar.py Outdated Show resolved Hide resolved

glemaitre mentioned this pull request Jan 13, 2025

feat: Leverage EstimatorReport in a CrossValidateReport #1014

Closed

sylvaincom reviewed Jan 17, 2025

View reviewed changes

skore/src/skore/sklearn/_cross_validation/report.py Outdated Show resolved Hide resolved

augustebaum suggested changes Jan 17, 2025

View reviewed changes

glemaitre and others added 2 commits January 17, 2025 15:15

Update skore/src/skore/sklearn/_cross_validation/report.py

b4523de

Co-authored-by: Sylvain Combettes <[email protected]>

Update skore/src/skore/sklearn/_cross_validation/report.py

9fa5f17

Co-authored-by: Sylvain Combettes <[email protected]>

glemaitre and others added 8 commits January 17, 2025 15:22

Update skore/src/skore/sklearn/_estimator/report.py

95478c6

Co-authored-by: Auguste Baum <[email protected]>

Update skore/src/skore/sklearn/_cross_validation/__init__.py

c8eb0a4

Co-authored-by: Auguste Baum <[email protected]>

chore: Harmonize message to note in notes-related methods (probab…

87c0d19

…l-ai#1143)

fix(UI): Item card actions are now aligned (probabl-ai#1145)

3f6be36

Closes probabl-ai#1110

fix(UI): Ellipsis long item name (probabl-ai#1147)

574cb65

Closes probabl-ai#1065 ![Screenshot 2025-01-17 at 14 58 49](https://github.com/user-attachments/assets/9a261b51-5d1f-4bb5-a383-8cf79d94c47f)

ci: Fix timeout by upgrading scikit-learn to latest bugfix (probabl-a…

c598b86

…i#1141) Co-authored-by: Thomas S. <[email protected]>

iter

94d43b4

glemaitre requested review from thomass-dev and rouk1 as code owners January 17, 2025 14:28

order matter

25bb9ba

thomass-dev reviewed Jan 17, 2025

View reviewed changes

.github/workflows/backend.yml Outdated Show resolved Hide resolved

glemaitre added 4 commits January 18, 2025 14:20

Merge remote-tracking branch 'origin/main' into cross_validation_repo…

3810c8d

…rt_3

fix: Cache the pos_label=None when calling cache_predictions

2c68c9e

Merge branch 'bug_pos_label_none_cache' into cross_validation_report_3

fe6bfa0

iter

e989de0

glemaitre mentioned this pull request Jan 19, 2025

docs: Add example to showcase EstimatorReport and CrossValidationReport #1156

Merged

Merge branch 'main' into cross_validation_report_3

ccafc68

augustebaum approved these changes Jan 20, 2025

View reviewed changes

augustebaum mentioned this pull request Jan 20, 2025

docs(CrossValidationReport): Document behaviour of CrossValidationReport.cache_predictions #1162

Open

augustebaum merged commit a3edbc3 into probabl-ai:main Jan 20, 2025
13 of 18 checks passed

sylvaincom mentioned this pull request Jan 20, 2025

Skore UI / Activity feed: Smaller layout and thoughtful information display #1047

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Revamp CrossValidationReport to use EstimatorReport #1091

feat: Revamp CrossValidationReport to use EstimatorReport #1091

glemaitre commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

augustebaum commented Jan 13, 2025

glemaitre commented Jan 13, 2025

augustebaum left a comment

glemaitre commented Jan 17, 2025 •

edited by augustebaum

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading

glemaitre commented Jan 17, 2025

augustebaum left a comment

feat: Revamp CrossValidationReport to use EstimatorReport #1091

feat: Revamp CrossValidationReport to use EstimatorReport #1091

Conversation

glemaitre commented Jan 10, 2025 • edited Loading

TODO

github-actions bot commented Jan 10, 2025 • edited Loading

github-actions bot commented Jan 10, 2025 • edited Loading

augustebaum commented Jan 13, 2025

glemaitre commented Jan 13, 2025

augustebaum left a comment

Choose a reason for hiding this comment

glemaitre commented Jan 17, 2025 • edited by augustebaum Loading

github-actions bot commented Jan 17, 2025 • edited Loading

Coverage Report for frontend

glemaitre commented Jan 17, 2025

augustebaum left a comment

Choose a reason for hiding this comment

glemaitre commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 10, 2025 •

edited

Loading

glemaitre commented Jan 17, 2025 •

edited by augustebaum

Loading

github-actions bot commented Jan 17, 2025 •

edited

Loading